Bayesian Back-Propagation

Connectionist feed-forward networks, t rained with backpropagat ion, can be used both for nonlinear regression and for (discrete one-of-C ) classification. This paper presents approximate Bayesian meth ods to statistical components of back-propagat ion: choosing a cost funct ion and penalty term (interpreted as a form of prior probability), pruning insignifican t weights, est imat ing the uncertainty of weights, predict ing for new pat terns ("out -of-sample") , est imating the uncertainty in the choice of this predict ion ("erro r bars" ), estimating the generalizat ion erro r, comparing different network st ructures, and handling missing values in the t raining patterns. These methods extend some heurist ic techniques suggested in the literature, and in most cases require a small addit ional facto r in comput at ion during back-propagat ion, or computation once back-pro pagat ion has finished.